Repository navigation
Conversation
d327245 to
b5abb7e
Compare
|
We need not change the footer, but use the new nested footer stats + AggFns |
|
@joseph-isaacs yeap, I just cleaned it up. I already have the new AggFn, but I'm still refining it a bit |
|
Ok, so a bit of an overview, the options for the aggfn are like this: pub struct SortOrderOptions {
pub fields: Vec<(FieldName, SortDirection)>
}The sort uses I still have to do some checks around the writer and bench. |
U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and #11016, and the DataFusion literal fix #11067 merged right after U2. So order constraint 1 holds, and phase 1 starts from develop. The graph shows these nodes as merged, and parts 2 and 4 name the PR that fixed each finding. #10953 now keeps i32 days and widens only the days part to i64 when a value does not fit. It has no review, and it conflicts with the DateTimeParts compare kernel that U2 changed. #11056 still waits on its request for changes. #10661 replaces the closed #10482 for file sort statistics. In the type roadmap, only the N1 sentence changes. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…11114 Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sort keys indicate that a file is lexicographically sorted. The current implementation writes and reads sort keys present in a footer. One currently missing piece is that it doesn't assert that the rows are truly sorted, so a user could write an unsorted file and declare in the footer that the rows are sorted. To add this assertion, we could use an agg_fn that checks multiple columns at the same time and validates that the rows are sorted across all chunks. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Unintended changes. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
When reviewing the code, I noticed that footer properties are always vectors, so I stopped to think about whether a vec of sort keys would make sense, and it turns out that yes, even DataFusion can support multiple sort keys. So what I did was just update it from a single sort key to multiple valid sort keys. Also added SortKey and SortKeyRef as type aliases. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
The previous work was incorrectly writing the sort keys in the footer as a spec rather than writing them as an aggregate function. This commit cleans up the previous incorrect work. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
This commit adds the SortOrder aggregate fn into the `vortex-array` crate. The new `SortOrder` basically reuses the sort already implemented for fastpaths, so it has the same limitations (only ASC and nulls first). The `SortOptions` are ser/de using protobuf, and there is no support for strictness, but I could simple be added. I tried to add enough documentation to understand how the pair check works. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
I think the most important to measure is for utf-8 and the single column bench to assert that not much on top of the fastest case (single column/field sort), similar to `IsSorted`. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Similar to other new agg fns, they need to be added to the unstable edition. Also, this commit adds a test to check that the aggregate fn is correctly written into the file stats, but it is not present in the layout zones. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Given that already exists `IsSorted` and `IsStrictSorted` I found appropiate to call it `IsCompoundSorted`. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
4d6d6a5 to
51f6866
Compare
There was one pending TODO about where to check whether the root entry was compatible with the `IsCompoundSorted` agg fn. This is just a fix for that, the check now lives in `FileStatsAccumulator::try_new`. If the root has no entry for the stats, the write should err. Otherwise the stat won't be written and the user won't notice. So, if the user requests `IsCompoundSorted` with legacy stats, or with a non-nullable root (which has no root entry), the write now errs. Signed-off-by: Joaquin Colacci <joaquincolacci@gmail.com>
|
@joseph-isaacs, rebased and also fixed a |
| // Nested stats have a root entry, legacy stats don't. | ||
| // This is only true if the root is nullable. If it is non-nullable, | ||
| // the nested stats don't have a root entry either. | ||
| let has_root_entry = write_nested_stats && dtype.is_nullable(); |
There was a problem hiding this comment.
Just saw a related issue (#9205) and wanted to just leave a heads up that the agg fn can only be written when the root is nullable. So this will probably be more usable after that gets fixed, or if we get root-level stats support for non-nullable roots.
Adds a new agg fn to check a compound sort key and store the result in the file stats.
Summary
The current implementation adds a new aggregate function called
IsCompoundSortedthat lets the user check whether rows are sorted by a declared compound key. It uses the same sort order asIsSorted, so it has the same limitations (onlyASCwith nulls first.)Changes
IsCompoundSortedagg fn and its optionsAPI Changes